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1.  Introduction 


Battlefield  weather  forecasting  has  transitioned  from  a  human  forecaster  located  in 
theater  to  the  usage  of  computerized  systems  that  include  Numerical  Weather 
Prediction  (NWP)  models,  with  the  human  forecaster  being  far  from  the  area  of 
interest.1  Thus,  the  accuracy  of  these  models  is  becoming  more  important  on  the 
battlefield.  The  US  Army  Research  Laboratory  (ARL)  modelers  are  using  their  own 
high-resolution  configuration  of  the  Weather  Research  and  Forecasting  (WRF)2 
model,  referred  to  as  Weather  Running  Estimate-Nowcast  (WRE-N),3  to  produce 
and  improve  the  accuracy  of  weather  forecasts. 

Nowcast  model  assessment  involves  applying  the  model  verification  techniques  to 
generate  error  statistics  to  characterize  model  performance  and  provide  insight  on 
how  to  improve  the  accuracy  of  the  forecasts  produced  by  the  ARL’s  nowcast 
model,  WRE-N.4  ARL  has  implemented  Model  Evaluation  Tools  (MET)  developed 
by  the  National  Center  for  Atmospheric  Research  (NCAR)  that  provide  multiple 
tools  to  characterize  domain-level  model  performance.  MET  was  developed  at 
NCAR  through  a  grant  from  the  United  States  Air  Force  557th  Weather  Wing 
(fonnerly  the  Air  Force  Weather  Agency),  where  NCAR  is  sponsored  by  the  United 
States  National  Science  Foundation.  During  the  assessment  process,  the  model’s 
output  and  the  weather  observations  (or  ground  truth)  are  used  by  the  MET 
software.  MET  compares  the  weather  observation  data  with  the  WRE-N  model 
forecast  output  to  calculate  various  error  statistics. 

The  2  main  goals  of  generating  error  statistics  are  to,  first,  provide  guidance  to  the 
warfighters  about  the  accuracy  of  the  forecast  models  and,  secondly,  provide 
modelers  with  the  information  needed  to  understand  the  model  errors  and  how  their 
algorithm  changes  might  mitigate  these  errors.  In  doing  so,  the  model  output  and 
error  statistics  from  the  original  model  can  be  quantitatively  compared  to  the  model 
output  and  error  statistics  from  the  upgraded  model  that  includes  the  modeler’s 
algorithm  changes.  By  performing  this  comparison,  the  modeler  can  easily 
detennine  which  module  or  physics  algorithm  is  affecting  the  model’s  forecast  in  a 
way  that  reduces  the  size  of  the  errors.  Thus,  developing  an  improved  forecast 
model  will,  in  turn,  provide  better  forecast  data  for  ingesting  into  decision  support 
tools  that  can  be  later  transitioned  to  the  Warfighter  on  the  battlefield. 

In  this  document,  a  process  for  automating  the  running  of  the  MET  statistical 
verification  tool,  Point-Stat  (version  4.1), 5  was  implemented  to  generate  domain- 
level  error  statistics  for  the  ARL  modelers  who  are  actively  trying  to  improve  the 
WRE-N  model  forecasts.  The  Point-Stat  tool  calculates  traditional,  grid-to-point 
error  statistics  by  calculating  the  forecast  minus  observation  differences.  The 
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statistics  derived  from  these  differences  are  output  as  statistical  text  files.6  The 
previous  implementation  of  the  Point-Stat  tool  required  the  model  forecast  and 
observation  data  to  be  processed  for  one  run  configuration  at  a  time;  that  is,  only 
one  model  run,  domain,  hour,  and  case  study  date  could  be  set  for  each  run.  This 
process  was  time-consuming  when  multiple  configurations  were  required  by  the 
user.  Also,  each  run  of  the  tool  overwrote  the  data  from  the  previous  run.  The 
automation  process  of  implementing  the  Point-Stat  tool,  referred  to  as  Point-Stat 
Automation  (PSA),  was  needed  to  generate  error  statistics  efficiently  for  multiple 
model  runs,  domains,  hours,  and  dates.  In  addition,  this  process  provided  a 
controlled,  automatic  data  structure,  which  uniquely  identified  each  dataset  of  error 
statistics  that  will  be  easily  used  in  future  data  analysis  by  ARL  modelers. 

2.  Development  Environment 

The  automation  of  Point-Stat  processes  (i.e.,  PSA)  was  developed  using  Python 
3.5.*  Python  was  selected  because  it  is  easy  to  use,  widely  used  for  scripting,  and 
satisfies  all  the  requirements  to  automate  the  implementation  of  the  Point-Stat  tool. 
In  addition,  the  Weather  Running  Estimate-Nowcast  Real  Time  (WREN  RT) 
system  currently  in  development  is  implemented  in  Python.  WREN  RT  will  be  a 
system  that  will  automate  real-time  WRE-N  model  simulations,  collect  and  quality 
control  check  weather  observations  for  assimilation  and  verification,  and  produce 
user-friendly  graphical  output  of  weather  forecasts  as  well  as  verification  statistics, 
such  as  the  error  statistics  output  generated  from  PSA.  The  ultimate  goal  will  be  to 
integrate  WREN  RT  with  the  automation  Point-Stat  process  and  scripts  described 
in  this  document,  so  that  error  statistics  can  be  generated  “on  the  fly”  after  a  WRF 
or  WRE-N  model  run  is  completed.  Initial  implementation  is  envisioned  to  perform 
this  task  over  domains  centered  near  White  Sands  Missile  Range,  New  Mexico, 
where  the  Meteorological  Sensor  Array  (MSA)  will  be  located.  The  MSA  will 
provide  additional,  high-resolution  observations  for  the  verification  of  the  WRE-N 
model.  Having  both  PSA  and  WREN  RT  in  Python  will  bring  consistency  to  the 
whole  automation  process  and  make  integration  easier  in  the  future. 

3.  Description  of  Scripts 

The  automation  process  of  Point-Stat  involved  the  development  of  the  following  2 
Python  scripts:  read_configFiles.py  and  runPointStat.py.  The  first  script, 
read_configFiles.py,  mainly  handles  the  reading  and  parsing  of  each  configuration 
parameter  set  in  the  PSA  configuration  file,  ps auto  config,  and  the  WREN_RT 
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configuration  file,  wren.config  (see  Section  5).  All  the  configuration  values  are 
stored  into  arrays  and  passed  as  arguments  to  the  second  script,  runPointStat.py, 
which  handles  the  implementation  of  running  the  Point-Stat  tool  using  the 
configuration  values.  To  run  the  script,  the  user  must  set  the  configuration  values 
in ps_auto_config  and  wren_rt.config  files  to  satisfy  their  unique  data  requirements. 
Then,  the  user  executes  the  following  command,  “python  runPointStat.py”,  to 
automatically  calculate  the  domain-level  error  statistics.  During  the  script’s 
execution,  the  script  makes  repetitive  command  calls  to  the  Point-Stat  tool  in  order 
to  generate  all  the  error  statistics  output  based  on  the  user’s  configuration  specified 
in  the  ps_auto_config  file. 

4.  Data  Structure 


The  PSA  scripts  use  input  and  output  data  that  are  organized  in  a  particular 
structure.  The  input  data  specified  by  the  scripts  consist  of  forecast  data  and 
observation  data  (see  Section  6),  as  well  as  Point-Stat  domain-level,  PSA,  and 
WREN  RT  configuration  files  (see  Section  5).  The  standard  data  structure  for  the 
PSA  process  is  depicted  in  Fig.  1.  The  forecast  data,  observation  data,  and  Point- 
Stat  domain-level  configuration  files  are  the  input  data  required  to  run  the  Point- 
Stat  tool  (i.e.,  data  in  the  <forecast_dir>,  <obs_dir>,  and  <config_dir>  directories, 
respectively,  as  shown  in  Fig.  1),  while  the  PSA  and  WREN_RT  configuration  files 
are  required  input  data  by  the  PSA  scripts  (i.e.,  ps_auto_config  and  wren.config, 
respectively,  as  shown  in  Fig.  1).  The  Point-Stat  configuration  files  specify  the 
types  of  statistical  output  to  be  generated.  The  output  data  consist  of  the  Point-Stat 
error  statistics  and  PSA  log  file  (i.e.,  data  in  the  <output_dir>  and  <log_dir>, 
respectively,  as  shown  in  Fig.  1).  The  error  statistics  output  consists  of  the 
calculated  error  statistics  that  are  generated  from  the  Point-Stat  tool  (see  Section  9). 
The  Point-Stat  log  file  is  generated  by  the  Point-Stat  tool  during  execution  and 
displays  log  and  debug  information  (see  Section  8).  The  PSA  log  file  is  similar  to 
the  Point-Stat  log  file,  but  it,  instead,  contains  log  and  debug  infonnation  relating 
to  the  PSA  execution. 
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<forecast dir> 

<date><modeler name> 

<forecast  str> 

-  WRF/WRE-N  model  output  files 

J  <obs_dir> 

<date><m  odeler nam  e> 

i— —  Observation  NetCDF  files  (.nc) 

<config dir> 

PointStatConfig_m<model#>o<domain#>_<modeler_name>_ 

<config  str>  <assessment  option> files 

<log dir> 


<date>_<H  H>_m  <m  od  el#>o<do  main#>_<mod  eler_na  me  >_ 
<log_str>_<assessment_option>_<date>_<HH>Jog  files* 


<Reference  Directory> 


<output dir> 


results_m  <model#>o<domain#>_ 
<modeler_name>_<output_str>_ 
<assessment_option>_<date>_<HH>* 


<date> 


MET  Point-Stat  output/ 
error  statistics  files 


ps_auto_config 


ps_a  uto_lo  g_<d  ate>_<HH  >* 
read_configFiles.py 
runPointStat.py 
wren.config 


Directory  -  Bold 
File  -  Not  Bold 
<date>  -  YYYYMMDD 
<HH>  -  two-digit  military  hour 
model#  -  any  number  from  1 
domain#  -  any  number  between  1  to  7 

*<date>_<HH>  -  current  date  and  hour 
of  processing 


Fig.  1  Standard  data  structure  of  PSA 

5.  Configuration  Files 

The  PSA  configuration  file,  ps_auto_config,  controls  the  process  and  output  of  the 
PSA  scripts.  All  the  values  set  in  the  PSA  configuration  file  determine  how  the 
scripts  use  the  Point-Stat  tool  to  generate  error  statistics.  A  description  of  all  these 
configuration  values  can  be  found  in  Table  1.  A  template  of  ps_auto_config  is 
provided  along  with  the  PSA  scripts  and  the  user  can  modify  it  according  to  their 
specific  needs.  An  example ps_auto_config  file  and  its  corresponding  data  structure 
is  shown  in  Figs.  2  and  3,  respectively. 

The  WRENRT  configuration  file,  wren.config,  contains  configuration  settings  or 
properties  from  the  WRE-N  model  runs.  Currently,  the  PSA  scripts  map  only  the 
configuration  parameter,  <horizontal_grid_spacing_m>,  in  the  file  to  retrieve 
values  of  the  model  grid  resolutions.  For  example,  as  shown  in  Fig.  4,  ml,  m2,  and 
m3  would  be  mapped  to  27000.0  (27  km),  9000.0  m  (9  km),  and  3000.0  (3  km), 
respectively.  Note  that  once  the  integration  with  the  WREN_RT  system  is 
complete,  this  file  will  be  auto-generated  or  transferred  by  the  system,  so  that  error 
statistics  can  be  developed  “on-the-fly”  after  a  WRE-N  model  run. 
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Table  1  PSA  configuration  values  in  psautoconfig 


Configuration 

Parameter 

Description 

model 

ml,  m2,  m3,  m4. .  ,m<n>,  or  ALL 

One  or  more  model  runs  separated  by  a  comma  “  ”  or  All  model  runs.  There  can  be  any  number  of 
model  runs  denoted  by  m<n>  starting  at  1.  The  value  of  “ALL"  represents  all  model  runs  that  arc 
available.  It  will  search  the  forecast  data  directory  (i.e.,  <forecast_dir>)  and  determine  which  model 
runs  are  available  for  processing. 

domain 

1)1  (ol).  1)2  (o2),  1)3  (o3),  1)4  (o4),  1)5  (c5),  1)6  (06),  1)7  (o7),  or  ALL 

One  or  more  domains  separated  by  a  comma  or  All  domains.  The  maximum  number  of  domains  is 

7.  The  value  of  “ALL"  represents  all  domains  that  are  available.  It  will  search  the  forecast  data 
directory  (i.e.,  <forccast  dir>)  and  determine  which  domains  arc  available  for  processing. 

hours 

HH:MM:SS  or  .ALL 

One  or  more  hours  or  times  separated  by  a  comma  or  All  hours.  The  HH:MM:SS  specifics  the 
format  of  the  time  listed  as  hours,  minutes,  and  seconds  respectively  in  Coordinated  Universal  Time 
(UTC),  such  as  1 2:00:00.  The  value  of  “ALL"  represents  all  hours  that  arc  available.  It  will  search  the 
forecast  data  directory  (i.e ,  <forecast  dir'--)  and  determine  which  hours  are  available  for  processing 

date 

YYYYMMDD 

One  or  more  case  study  dates  separated  by  a  comma  The  Y  Y  Y  YMMDD  specifies  the  format  of  the 

case  study  date  listed  as  year,  month,  and  day  respectively,  such  as  20160801  for  August  01,  2016. 

modelername 

The  name  of  modeler 

One  or  more  modeler’s  names  separated  by  a  comma  Note  that  if  using  multiple  modelers  for  one 
run,  the  configuration  parameters  above  should  be  exactly  the  same  Otherwise,  it  is  advised  to  create 
different  configuration  files  for  each  modeler 

forecastjdir 

Directory  path  where  the  WRF  forecast  input  data  resides  in  relation  to  the  PSA 
scripts 

The  scripts  searches  for  the  forecast  data  using  the  following  naming  convention: 

< forecast  tSr>/<date>  <modeler  name >  <forecast  str > 

User-defined  string  to  assist  in  identifying  the  forceast  data 

obs_dir 

Directory  path  where  the  point  observation  input  data  (i.e.,  NetCDF  files)  resides  in 
relation  to  the  PSA  scripts 

The  scripts  searches  for  the  observation  data  using  the  following  naming  convention: 

:  obs  dir-date-  modeler  name >  obs  str> 

obs  str 

User-defined  string  to  assist  in  identifying  the  point  observation  data 

conjigdir 

Directory  path  where  the  Point-Stat  configuration  files  reside  in  relation  to  the  PSA 
scripts 

The  scripts  search  for  the  configuration  files  using  the  following  naming  convention: 
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Table  1  PSA  configuration  values  in  psautoconfig  (continued) 


':co«^g_rfi>>/'PointSlatConfig  nv  model#  o-  domain#  -modeler  name''  ’:config_str> 

-assessment j>ption> 

User-defined  string  to  assist  in  identifying  the  configuration  files 

outputdir 

Directory  path  where  the  MET  Point-Stat  output  or  error  statistics  data  will  reside  in 
relation  to  the  PSA  scripts 

The  scripts  create  the  output  director)'  using  the  following  nam  ing  convention : 

‘-output_dir> /lesulls  m-  model#  o-  domain#  •  modelername-  ’  output _str~' 

■  assessment_option>  <YYYYMMDD>  <HH >l<date> 

Note  that  <YYYYMMDD>  <HH>  represents  the  current  date  and  military  hour  of  processing. 

User-defined  string  to  assist  in  identifying  the  MET  Point-Stat  output  data 

log  dir 

Directory  path  where  the  MET  Point-Stat  log  files  will  reside  in  relation  to  the  PSA 
scripts 

The  scripts  creates  the  log  directory  using  the  following  naming  convention: 

<log_dir>l<date>  <hour>  m<model#>o<domain#>  -modeler  name >  <log_str> 
<awmmen/_«pfi'on>_<YYYYMMDD>_<HH>_log 

Note  that  <  Y  Y  Y  Y MMDD>  <HH>  represents  the  current  date  and  m  ilitary  hour  of  processing. 

log  str 

User-defined  string  to  assist  in  identifying  the  MET  Point-Stat  log  files 

log_verbosity 

Verbosity  level  in  Point-Stat  log  files 

Setting  the  verbosity  to  0  will  make  the  tool  run  with  no  log  messages,  while  increasing  the  verbosity 
above  1  will  increase  the  amount  of  logging. 

executeoption 

Execution  option  for  the  PSA  scripts  to  run  manually  or  on  a  scheduled  basis 

To  run  the  scripts  manually,  set  the  option  to  0. 

*To  run  on  a  scheduled  basis,  set  the  option  to  1  and  the  case  study  date  will  be  set  to  the  next  day  of 
case  study  date.  The  script  will  run  on  a  scheduled  basis  based  on  data  availability.  Note  that  the  next 
day  of  the  case  study  date  is  used  because  the  verification  or  assessment  cannot  be  done  until  the  wall 
clock  time  is  past  the  end  of  the  time  penod  of  the  model  forecasts. 

debug 

Debug  option  to  display  debug  messages  during  PSA  execution 

To  turn  the  debug  mode  off,  set  the  option  to  0.  No  debug  information  is  displayed  on  the  screen  during 
the  execution  of  the  scripts. 

To  turn  the  debug  mode  on,  set  the  option  to  1.  When  the  debug  mode  is  turned  on,  the  script  will 
display  the  debug  information  on  the  screen  during  the  execution  of  the  scripts. 

Note  that  the  Point-Stat  auto  log  file,  ps  auto  Jog,  will  always  contain  debug  and  log  information 
regardless  whether  the  debug  mode  is  on  or  off. 

msessmentopdon 

Assessment  option  to  perform  Domain  Dependent  Assessment  (DDA)  or  Resolution 
Dependent  Assessment  (RDA) 

To  set  DDA,  set  the  option  to  0  (eg,  ml ol,  m2o2,  m3o3) 

To  set  RDA  set  the  option  to  1  (e  g,  mlo3,  m2o3,  m3o3) 

This  is  a  future  capability  and  will  be  available  once  the  integration  with  WREN_RT  is  complete. 
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#PS_AUTO_CONFIG  Settings 

model  =  ALL 

domain  =  ALL 

hours  =  12:00:00 

date  =  20120207 

modelername  =  P 

forecastdir  =  METWRFpostprd 

forecasts  tr  =  test 

obs_dir  =  MET_obs/little_r 

obs_str  =  test 

conflgdir  =  METPointStat/configs 

configstr  =  test 

outputdir  =  METPointStat 

outputstr  =  test 

log_dir  =  MET_PointStat/logs 

log_str  =  test 

log_verbosity  =  5 

executeoption  =  0 

debug  = 1 

assessment  option=0 


Fig.  2  An  example  of  PSA  configuration  fil e,  ps_auto_config 


<Reference  Directory> 


— MET_WRFpostprd -  20120207_P_test 


MET  obs 


20120207  P  test 


MET  PointStat 


configs 


logs 


WRF/WRE-N  model  output  files 


Observation  NetCDF  files  (.nc) 


PointStatConfig_m<model#>o<domain#>_P_test_ 
<assessment_option>  files 


20 1 2020  7_<H  H>_m  <m  ode  l#>o<dom  ain#  >_P_test_ 
<assessment_option>_<date>_<HH>_log  files* 


results_m  <model#>o<domain#>_ 

P_test_<  a  s  s  e  s  s  m  ent_opti  o  n  >_ 
<date>  <HH>* 


ic\Mf\ir\7  _ MET  Point-Stat  output/ 

zunuzur  error  statistics  files 


ps auto config 

_  ps_auto_log_<date>_<HH>* 
-  read_configFiles.py 
_  runPointStat.py 
wren.config 


Directory  -  Bold 
File -Not  Bold 
<date>  -  YYYYMMDD 
<HH>  -  two-digit  military  hour 
model#  -  any  number  from  1 
domain#  -  any  number  between  1  to  7 

*<date>_<HH>  -  current  date  and  hour 
of  processing 


Fig.  3  PSA  data  structure  based  on  the  example  configuration  file 
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#WREN_RT  Settings 

horizontalgridspacingm  =  27000.0,  9000.0,  3000.0 


<Other  WREN  RT  Property  Settings> 


Fig.  4  An  example  of  WREN_RT  configuration  file,  wren.config 

6.  Forecast  and  Observation  Data 


The  input  data  of  the  PSA  scripts  are  gridded  WRF  forecast  files  and  point 
observations  required  by  the  Point-Stat  tool.  The  WRF  forecast  data  should  be 
organized  by  the  case  study  date,  modeler’s  name,  and  user-defined  forecast  string, 
as  described  in  Section  4.  The  forecast  files  are  grib  files  created  by  the  National 
Centers  for  Environmental  Prediction  (NCEP)  Unified  Post  Processor  (UPP).7 
There  should  also  be  a  forecast  file  corresponding  to  each  case  study  date,  domain, 
and  hour  that  the  user  wants  to  generate  error  statistics  from  the  Point-Stat  tool.  For 
example,  if  the  case  study  date,  domain,  and  hours  are  2016-08-01,  Dl,  and 
12:00:00Z  to  6:00:00Z  (next  day),  respectively,  then  there  will  be  18  gridded  WRF 
forecast  files  in  the  forecast  data  directory.  As  stated  in  Section  5,  the  location  of 
the  forecast  data  should  be  set  in  the  ps_auto_config  file  by  using  the  configuration 
parameter,  <forecast_dir>,  before  executing  the  PSA  scripts.  The  point  observation 
files  are  in  the  MET-specific  NetCDF  format.  Similar  to  the  WRF  forecast  files, 
the  point  observations  should  be  organized  by  the  case  study  date,  modeler’s  name, 
and  user-defined  observation  string,  as  described  in  Section  4.  Note  that  the  PSA 
scripts  use  the  observation  data  for  the  domain  currently  being  assessed  or  verified, 
if  it  is  available  in  the  observation  data  directory.  Otherwise,  the  observation  data 
for  the  outermost  (largest)  domain,  Dl,  is  used. 

7.  Domain  vs.  Resolution  Dependent  Assessment 

The  PSA  scripts  can  perform  the  following  2  types  of  assessments:  Domain 
Dependent  Assessment  (DDA)  and  Resolution  Dependent  Assessment  (RDA). 
This  is  set  in  the  configuration  file,  ps_auto_config,  by  the  user  through  the 
configuration  parameter,  <assessment_option>,  as  described  in  Table  1. 

The  PSA  scripts  use  DDA  to  generate  error  statistics  using  Point-Stat  for  multiple 
domains,  starting  from  the  outermost  (largest)  domain  to  the  innermost  (smallest) 
domain.  The  outermost  domain  is  always  classified  as  Dl  (ol).  The  WRE-N  model 
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run  output  is  assessed  over  its  respective  domains.  For  example,  if  there  are  3 
domains  that  are  assessed  over  3  different  model  runs,  namely  27  km  (ml),  9  km 
(m2),  and  3  km  (m3),  all  centered  on  the  same  geographical  point,  then  the  27  km 
(ml),  9  km  (m2),  and  3  km  (m3)  will  be  evaluated  over  D1  (ol),  D2  (o2),  and  D3 
(o3),  respectively.  The  goal  would  be  to  assess  and  calculate  error  statistics  for  the 
3  model  domains  to  determine  how  they  perform  over  their  respective  domains  (i.e., 
mlol,  m2o2,  m3o3),  as  shown  in  Fig.  5. 


mlol 


m2o2 

m3o3 

Fig.  5  An  example  DDA  over  3  domains  and  model  runs 

The  PSA  scripts  use  RDA  to  generate  error  statistics  using  Point-Stat  for  multiple 
WRE-N  model  runs  over  only  the  area  covered  by  the  innermost  domain.  This 
allows  the  evaluation  of  any  statistical  differences  among  the  different  model  runs 
over  a  consistent  domain.  For  RDA,  the  innennost  domain  masking  files  are  needed 
and  should  be  placed  in  the  same  directory  as  the  Point-Stat  configuration  directory 
specified  in  ps_auto_config.  For  example,  if  there  are  3  domains  that  are  assessed 
over  3  different  model  runs — 9  km  (ml),  3  km  (m2),  and  1  km  (m3) — all  centered 
on  the  same  geographical  point,  then  9  km,  3  km,  and  1  km  will  all  be  evaluated 
only  over  the  innermost  domain,  D3  (o3).  The  goal  would  be  to  assess  and  calculate 
error  statistics  on  the  3  model  runs  to  determine  how  they  perfonn  over  the 
innennost  domain,  D3  (i.e.,  mlo3,  m2o3,  m3o3). 

8.  Log  File 

After  the  execution  of  the  PSA  scripts,  a  log  file  is  created  that  contains  all  the  log 
and  debug  information  pertaining  to  a  particular  PSA  run.  The  PSA  log  file  is 
produced  and  stored  in  the  same  directory,  where  the  PSA  scripts  are  located,  as 
shown  in  Fig.  1.  An  example  of  a  PSA  log  file,  ps_auto_log,  is  provided  in  Fig.  6. 
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As  shown  in  Fig.  6,  a ps_auto_Iog  file  was  created  using  DDA,  one  case-study  date 
(i.e.,  20120207),  3  domains,  3  model  runs,  and  1  h  (i.e.,  12:00:00Z)  after  a 
successful  run  of  the  PSA  scripts.  The  naming  convention  of  the  log  file  is  as 
follows:  ps_auto_log_<YYYYM M D D>_<H H >,  where  <YYYYMMDD>  is  the 
current  date  of  processing  and  <HH>  is  the  military  hour  of  processing.  The  log  file 
is  created  and  stored  in  the  same  directory  where  the  PSA  script,  runPointStat.py, 
is  executed.  The  log  file  includes  various  log  information,  such  as  the  MET  version 
number,  assessment  option,  modeler’s  name,  model  number,  domain  number,  hour, 
case  study  date  for  each  Point-Stat  command,  and  the  contents  of  its  command 
during  execution.  If  there  are  any  error  messages  from  executing  the  PSA  script, 
the  ps  auto  log  file  will  include  them.  If  any  issues  occur  relating  to  the  Point-Stat 
tool,  the  tool’s  log  file  will  include  them,  which  will  be  located  in  the  log  directory 
specified  in  the  ps_auto_config  file. 


***  Model  Evaluation  Tools  (METV4. 1)  *** 

Reading  the  Point-Stat  auto  configuration  file,  psautoconfig... 

Reading  the  WREN  RT  configuration  file,  wren.config... 

Setting  the  configuration  parameters... 

Running  for  Domain  Dependent  Assessment  (DDA)... 

Forecast  Data  File:  MET_WRFpostprd/20120207_P_FDDA/131  127A_WRFPRS_d01_2012-02-07_l 2:00:00 
Observation  Data  File:  MET  ohs/little  r/20 120207  P  FDDA  qc  obs  used  earth  relative.d01.2012-02- 
07_  12:00:00. OOOO.nc 

Created  output  directory:  METPointStat/rosultsml  ol_P_tcst_DDA_20 1 60726  1 4/20120207 
Running  PointStat  for  P  -  mlol,  15.75km  res  on  20120207  at  12:00:00Z 

point _stat  M ET_WRFpostprd/20 1 20207_P_FDDA/ 13112 7A_WRFPRS_dO  1_20 1 2-02-07_l  2 :00:00 
MET_obs/little_r/20 1 20207  P  FDDA/qc_obs_used_earth_relative.dO  1 .20 1 2-02-07_l  2:00:00.0000.nc 
MET  PointStat/configs/PointStatConfig  mlol  P  test  DDA-outdir 
MET_PointStat/results nil ol_P_test  DDA_201 60726_14/20 1 20207  -log 
MET  PointStat/logs/20 120207_1 2  mlol  JMest  DDA„20160726_14Jog  -v  5 

Forecast  Data  File:  MET  WREpostprd/201 20207  P  FDDA/ 131  127A  WRFPRS  dOl  2012-02-07  13:00:00 
C )bservation  Data  File  MET  ol>s  little  r  20 1 20207_P_FDDA  qc  obs  iised  earlh  relative.dOl  .201  2-02- 
07_  13:00:00. 0000.  nc 

Created  output  directory:  MET_PointStat/results_m2o2_P_test_DDA_20 1 60726_1 4/20120207 
Running  PointStat  for  P  -  m2o2,  5.25km  res  on  20120207  at  12:00:00Z 

point_stat  MET_WRFpostprd/20120207_P_FDDA/l31 127A_WRFPRS_d02_2012-02-07_l 2:00:00 
MET  obs/little  r/20 1 20207  P  FDD A/qc_obs_used_eaith_relative. d0 1 .20 1 2-02-07  1 2:00:00. OOOO.nc 
MET  PointStat. 'contias 'PointStatConfig  m2o2  P  test  DDA-outdir 
MET_PointStat/results_ni2o2_P_test_DDA_20 1 60726_ 14/20 1 20207  -log 
MET_PointStat/lqgs/20 1 20207  1 2_m2o2_P_test_DDA_20 1 607261 4_log  -v  5 

Forecast  Data  File:  MET_WRFpostprd/20120207_P_FDDA/131  127A_WRFPRS_d02_2012-02-07_l 3:00:00 
Observation  Data  File:  MET  obs/little  r/20120207  P  FDDA/qc  obs  used  earth  relative  dOl  20 12-02- 
07  1 3:00:00. OOOO.nc 

Forecast  Data  File:  MET_WRFpostprd/20120207_P_FDDA/131 127A_WRFPRS_d03_2012-02-07_12:00:00 
Observation  Data  File  Miff  obs  little  r 2012020“  P  I  DDAqc  obs  used  earth  relative. dOl  .201 2-02- 
07  1 2:00:00. OOOO.nc 

Created  output  directory:  MET_PointStat/results_m3o3_P_test_DDA_20 1 607261 4/20120207 
Running  PointStat  for  P  -  m3o3,  1.75km  res  on  20120207  at  12:00:00Z 

pointstat  M ET_WRFpostprd/20 1 20207  P  FDDA/ 1 3 1 1 27A_WRFPRS_d03_20 1 2-02-07  1 2 :00:00 
MET_obsdittle_r/201  20207_P_FDDA/qc_obs_used_earth_relativ'e.d01 .201  2-02 -071  2:00:00. OOOO.nc 
M  ET_  Point  St  at/  con  figsPoi  nt  St  at  Con  fig  m3o3_P_test_DDA  -outdir 
MET_PointStat/rcsults_m3o3_P_tcst_DDA_20 1607261 4/20120207  -log 
M ET_Point Stat/logs/ 20 1 20207_1 2_m3o3_P_tcst_DDA_20 1 60726_  1 4_log  -v  5 


Fig.  6  An  example  PSA  log  file, ps  auto  log,  using  DDA,  one  case-study  date,  3  domains,  3 
model  runs,  and  1  h 
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9.  Output  of  Error  Statistics 


After  the  execution  of  the  PSA  scripts,  the  error  statistics  results  are  automatically 
produced  and  stored  in  the  output  directory  specified  by  the  user’s  configuration  in 
the  ps_auto_config  file,  as  described  in  Table  1.  The  data  are  organized  by  model 
number,  domain  number,  modeler’s  name,  user-defined  output  string,  assessment 
option,  and  case  study  date,  as  shown  in  Fig.  1.  In  addition,  the  current  date  and 
military  hour  of  processing  are  appended  to  the  end  of  each  results’  directories  in 
format  of  YYYYMMDD  HH.  The  organization  of  the  data  prevents  the  output 
files  from  being  overwritten  if  the  scripts  are  executed  again  with  the  same 
configuration  parameters. 

The  output  of  the  Point-Stat  tool  consists  of  multiple  American  Standard  Code  for 
Infonnation  Interchange  (ASCII)-encoded  text  files  containing  a  summary  of 
calculated  error  statistics.  For  current  ARL  model  assessment  purposes,  there  are 
10  sets  of  text  files  for  each  case  study’s  date  and  hour,  where  there  is  one  file  each 
representing  continuous  statistics  (CNT),  contingency  table  counts  (CTC), 
contingency  table  statistics  (CTS),  forecast,  hit,  observation  rates  (FHO),  multi¬ 
category  contingency  table  counts  (MCTC),  multi-category  contingency  table 
statistics  (MCTS),  matched  pair  data  (MPR),  scalar  partial  sum  (SL1L2),  and  vector 
partial  sum  (VL1L2),  and  a  tenth  one  for  the  STAT  file  (.stat).5  The  STAT  file 
contains  all  of  the  error  statistics,  while  the  other  files  contain  identical  data,  but 
sorted  into  files  by  line  type. 

The  STAT  file  produced  by  the  Point-Stat  tool  uses  the  following  naming 
convention: 

point_stat_<prefix>_<HHMMSS>L_<YYYYMMDD>_<HHMMSS>V.stat, 

where  <prefix>  indicates  the  user-defined  string  defined  in  the  Point-Stat 
configuration  file,  <FIHMMSS>L  indicates  the  forecast  lead  time  and 
<YYYYMMDD_HHMMSS>  indicates  the  forecast  valid  time. 

Similarly,  the  output  of  the  other  text  files  are  named  as  follows: 

point_stat_<prefix>_<HHMMSS>L_<YYYYMMDD>_<HHMMSS>V_<type> 

.txt, 

where  <type>  is  one  of  cnt,  etc,  cts,  fho,  mete,  mcts,  mpr,  si  112,  or  vlll2  that 
indicates  the  line  type.5  Figure  7  shows  an  excerpt  from  the  beginning  of  an  FHO 
error  statistics  file  that  was  produced  by  the  PSA  scripts, 
point_stat_mlolP_000000L_20120207_120000V_fho.txt.  This  is  similar  to  the  8 
other  individual  statistics  files  that  are  produced  by  the  Point-Stat  tool. 
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VERSION  MODF.I.  FCST_I.EAD  FCSTVAI.IDBEG  FCST_VAI.ID_END  OBSJ.EAD  OBS_VAUD_BFG 
OBS  VAI.ID  END  FCST  VAR  FCST  I.KV  OBS  VAROBS  LEV  OBTYPH  VX  MASK  INTKRP  MTH1)  INTKRP  PNTS 
FCST  THRESH  OBS  l  HRESH  COV  THRESH  .ALPHA  LINE  TYPE  TOTAL  F  RATE  H  RATE  O  RATE 
V4.1  9kmPWRE-N  000000  20120207J20000  20 120207_1 20000  000000  20120207J  14000  201 20207J22000  TMP  P225- 
100  TMP  P225-100  ADPUPAFULL  DW  MEAN  4  >273.000  >273.000  ~NA  NA  FHO  99  0.00000 

0.00000  0.00000 

V4.1  9kmPWRE-N  000000  20120207  120000  20 120207_ 120000  000000  20120207  1 14000  20120207  122000  TMP  P225- 
100  TMP  P225-100  AIRCAR  FULL  DWMEAN  4  >273.000  >273.000  NA  NA  FHO  22  0.00000 

0.00000  0.00000 

V4.1  9kmPWRE-N  000000  20120207J20000  20 120207_1 20000  000000  20120207J 14000  20120207J22000  TMP  P425- 
225  TMP  P425-225  ADPUPA  FULL  DW  MEAN  4  >273.000  >273.000  ~NA  NA  FHO  153  0.00000 

0.00000  0.00000 

V4.1  9kinPWRE-N  000000  20 120207_1 20000  20120207_120000  000000  20120207J 14000  20120207_122000  TMP  P425- 
225  TMP  P425-225  AIRCAR  FULL  DW  MEAN  4  >273.000  >273.000  NA  NA  FHO  106  0.00000 

0.00000  0.00000 

V4.1  9kmPWRE-N  000000  20120207 J  20000  20120207J20000  000000  20120207  1 14000  20120207  122000  TMP  P625- 
425  TMP  P625-425  ADPUPA  FULL  DW  MEAN  4  >273.000  >273.000  NA  NA  FHO  127  0.00000 

0.00000  0.00000 

V4.1  9kmPWRE-N  000000  20120207J20000  20 120207_1 20000  000000  20120207J 14000  20120207J22000  TMP  P625- 
125  TMP  P625-425  AIRCAR  FULL  DW_MEAN  4  >273.000  >273.000  NA  NA  FHO  55  0.00000 

0.00000  0.00000 

V4.1  9kmPWRE-N  000000  20 120207J 20000  201 20207J  20000  000000  20120207_1 14000  20120207J22000  TMP  P775- 
625  TMP  P775-625  ADPUPA  FULL  DW_MEAN  4  >273.000  >273.000  NA  NA  FHO  106  0.27358 

0.19811  0.21698 

V4.1  9kmPWRE-N  000000  20120207  120000  20120207  120000  000000  20120207  1 14000  20120207  122000  TMP  P775- 
625  TMP  P775-62 5  AIRCAR  FULL  DW  MEAN  4  >273.000  >273.000  NA  NA  FHO  60  0.28333 

0.28333  0.31667 

V4.1  9kmPWRE-N  000000  20120207_120000  20 120207_1 20000  000000  20120207_1 14000  20120207  122000  TMP  P775- 
625  TMP  P775-625  ADPSFC  FULL  DW  MEAN  4  >273.000  >273.000  NA  NA  FHO  119  0.02521 

0.00000  0.09244 

V4.1  9kmPWRE-N  000000  20120207  120000  20120207  120000  000000  20120207  114000  20120207  122000  TMP  P875- 
775  TMP  P875-775  ADPUPA  FULL  DW  MEAN  4  >273.000  >273.000  NA  NA  FHO  71  0.85915 

0.83099  0.83099 

V4.1  9kmPWRE-N  000000  20 120207J  20000  20120207_120000  000000  20120207_1 14000  201 20207_122000  TMP  P875- 
775  TMP  P875-775  AIRCAR  FULL  DW  MEAN  4  >273.000  >273.000  NA  NA  FHO  37  1.00000 

0.97297  0.97297 

V4.1  9kinPWRE-N  000000  20120207J20000  20 120207_1 20000  000000  20120207J 14000  20120207_122000  TMP  P875- 
775  TMP  P875-775  ADPSFC  FULL  DW  MEAN  4  >273.000  >273.000  NA  NA  FHO  513  0.60039 

0.42495  0.45224 

V4.1  9killPWRE-N  000000  20120207J20000  20 1 20207J  20000  000000  20120207  1 14000  20120207J22000  TMP  P9I0- 
875  TMP  P91 0-875  ADPUPAFULL  DW  MEAN  4  >273.000  >273.000  ~NA  NA  FHO  18  1.00000 

1.00000  1.00000 

V4.1  9kmPWRE-N  000000  20120207J  20000  20120207  1  20000  000000  20120207J  14000  20120207J22000  TMP  P910- 
875  TMP  P910-875  AIRCAR  FULL  DW  MEAN  4  >273.000  >273.000  NA  NA  FHO  11  1.00000 

1.00000  1.00000 


Fig.  7  An  excerpt  of  a  Point-Stat  FHO  error  statistics  file  produced  by  the  PSA  scripts 


10.  Conclusion 


The  scripts  that  automate  the  Point-Stat  processes  (or  PSA)  were  developed  to 
easily  generate  domain-level  error  statistics  from  WRE-N  model  output  data  as 
soon  as  possible  following  the  model  run.  They  provide  an  automated,  efficient  data 
process  that  can  be  used  by  ARL  modelers.  This  simplifies  the  modeler’s 
performance  of  future  data  analysis  and  evaluation  of  WRE-N  model 
improvements. 

PSA  is  one  important  component  of  WREN  RT.  The  long-term  goal  is  to  have  the 
real-time  WRE-N  system  (i.e.,  WREN_RT),  that  is  currently  in  development, 
integrated  with  the  Point-Stat  automation  processes  (i.e.,  PSA)  described  in  this 
document.  When  using  the  WREN_RT  system,  the  following  tasks  will  be 
completed  in  an  automatic  process  using  the  following  order: 
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1.  Prepare  input  data  to  WRE-N,  including  collecting  the  ground-truth 
observations  and  perfonning  quality-control  checks  for  the  pre-forecast  data 
assimilation  period. 

2.  Run  the  WRE-N  model  to  generate  model  forecast  data. 

3.  Create  graphical  visualization  of  the  model  forecast  data. 

4.  Generate  error/verification  statistics  (such  as  from  PSA)  after  the  end  of  the 
forecast  period,  including  collecting  observations  and  performing  quality 
checks  for  the  entire  model  integration  period. 

5.  Populate  a  data  archive  containing  all  of  the  above  for  later  access  by  the 
ARL  modelers. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ARL 

US  Army  Research  Laboratory 

ASCII 

American  Standard  Code  for  Information  Interchange 

CNT 

Continuous  Statistics 

CTC 

Contingency  Table  Counts 

CTS 

Contingency  Table  Statistics 

DDA 

Domain  Dependent  Assessment 

FHO 

Forecast,  Hit,  Observation 

MCTC 

multi-category  contingency  table  counts 

MCTS 

multi-category  contingency  table  statistics 

MET 

Model  Evaluation  Tools 

MPR 

matched  pair  data 

NCAR 

National  Center  for  Atmospheric  Research 

NCEP 

National  Centers  for  Environmental  Prediction 

NWP 

Numerical  Weather  Prediction 

PSA 

Point-Stat  Automation 

RDA 

Resolution  Dependent  Assessment 

SL1L2 

scalar  partial  sum 

VL1L2 

vector  partial  sum 

WRE-N 

Weather  Running  Estimate-Nowcast 

WREN_RT 

Weather  Running  Estimate-Nowcast  Real  Time 

WRF 

Weather  Research  and  Forecasting 

UPP 

Unified  Post  Processor 

UTC 

Coordinated  Universal  Time 
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